A Dancing Programmer in an Immersive Virtual Environment

Our immersive programming system allows programs to be edited and controlled by direct manipulation and hand gestures in an immersive virtual environment utilizing multimodal interfaces. It lets beginners focus on learning the essence of programming without requiring familiarity with the keyboard and commands of a conventional editor. 1. Background It usually takes beginners a long time to make a program. This is partly because they need to understand the new concepts, syntax, and semantics of a programming language. However they often spend a lot of time editing a program and correcting typos and misspellings, according to our observation, when a text programming language is used because many novice programmers are not used to touch typing (especially in non-western countries) or the operations of a text editor or conventional integrated development environment (IDE) system. We think that this situation adversely affects not only their programming efficiency but also their understanding of the programming concepts. Multimodal interactions in immersive virtual environments will ameliorate the situation since the interactions can be better modeled after reality than a keyboard interface or GUI. Three-dimensional space should be used in order to fully utilize the body’s freedom of movement. To view objects in 3D, a 2D display is insufficient because it cannot give one an adequate sense of depth in 2D. Therefore a stereoscopic view is preferable to help one to touch, grasp, and move virtual objects in 3D. We have been developing an immersive programming system called ougi, which allows a programmer to edit and manipulate programs through direct manipulation and gestures using his/her hands in an immersive environment. The programmer appears to be dancing because he/she moves and gestures in the immersive virtual environment. The current implementation supports multimodal programming in a subset of the Java programming language although the system can be extended to support other languages. There has been considerable research on visual 3D programming and algorithm animation [2][4][5][8][9], or visual object-oriented languages [3]. However, a general-purpose object-oriented programming system for immersive environments has not been implemented. Moreover, interactive multimodal interfaces for programming in immersive environments have not been fully studied. Direct manipulation such as a grab-move-release (or get-and-put) operation is basic in 3D, but simply replacing the drag-and-drop operation in a GUI with a grab-move-release operation in 3D is insufficient. The design of 3D interactive interfaces is not a trivial problem. 2. Immersive Programming Although our system is an immersive and multimodal programming system, it does not completely eliminate textual representations. They are retained as useful because we have all learned written languages for many years and understood the meanings of many words. We are not sufficiently trained in understanding the meanings of graphical symbols such as icons. It is difficult to understand abstract concepts only from graphical symbols because we do not have standard symbols for abstract concepts. Since we are familiar with mathematical formulae, the usual mathematical expressions are also used in the system. It is reasonable to represent language elements by nested structures according to the programming language syntax. That representation is similar to nested boards or nested boxes such as [5]. However, the nested structures are not always shown in the ougi system since they may limit the visibility of the inside of the structure or become a visual clutter. Figure 1: Block statement with an incomplete expression. Figure 1 shows a block statement containing an incomplete expression without the nested structure being displayed. It looks like ordinary text, but has a hollow box as a placeholder that can include an expression or a variable. A virtual hand is putting a variable into the placeholder. When the virtual hand intersects with the regions of the expression, the nested structure of the expression is as shown in Figure 2. This reveals the syntax structure of Proceedings of the IEEE Symposia on Human-Centric Computing Languages and Environments (HCC’01) 0-7695-0474-4/01 $17.00 © 2001 IEEE that expression. Each region has a handle like a cylinder. Direct manipulation of the handle is used to move and copy the region. The handle is necessary for easy operations. If it were not used, the nested structures would have to be large enough to be separated from the outer and inner structures, otherwise it would be difficult to choose a middle structure without touching neighboring ones. The handle is also utilized to get and set properties. A hand gesture such as pinching and turning opens a panel for further interactions.